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Background / Context: 

The consequences of failing core academic courses during the first year of high school are dire. 
Research by the Consortium on Chicago School Research (CCSR) demonstrates that students 
who fall -off track” during the first year of high school have a substantially lower probability of 
graduating than students who stay -en track.” In the Chicago Public Schools (CPS), only about 
one-fifth of off-track freshmen — students who fail more than one semester of a core academic 
course and/or fail to earn enough credits to be promoted to 10th grade — graduate high school, 
compared with over 80% of on-track freshmen (Allensworth & Easton, 2005, 2007). Failure of 
Algebra I is particularly problematic. In CPS, only 13% of students who fail both semesters of 
Algebra I in 9th grade graduate in 4 years, and the largest share of 9th grade algebra failures 
occur in the second semester of the course. Elucidating the ways that students can get back on 
track is of the utmost policy importance. 

Credit recovery is one strategy to deal with high failure rates. The primary goal of credit 
recovery programs is to give students an opportunity to retake classes that they failed in an effort 
to get them back on track and keep them in school (Watson & Gemin, 2008). Most recently, as 
schools across the nation struggle to keep students on track and re-engage students who are off 
track, online learning has emerged as a promising and increasingly popular strategy for credit 
recovery: more than half of respondents from a national survey of administrators from 2,500 
school districts reported using online learning in their schools for credit recovery, with just over 
a fifth (22%) reporting -wide use” of online learning for this purpose (Greaves & Hayes, 2008). 

Despite the growing use of online courses for credit recovery, the evidence base is thin. This 
paper describes the design and initial implementation of a randomized control trial that was 
designed to strengthen the evidence base surrounding online courses used for credit recovery. 
They study is testing the (1) the impact of online Algebra I for credit recovery against the 
standard face-to-face (f2f) version of the course and (2) the effects of offering expanded credit 
recovery options with online algebra, relative to business as usual (i.e., the summer programming 
that schools would offer in the absence of efforts to expand credit recovery). 

Purpose / Objective / Research Question / Focus of Study: 

The primary goals and research questions of the study are: 

1) To test the efficacy of online Algebra I for credit recovery, compared with standard f2f 
Algebra I for credit recovery. Through a within-school, randomized trial in 20 schools with 
freshman Algebra I failure rates above 20%, we will directly compare online and f2f Algebra I in 
a head-to-head trial. This comparison will allow us to answer these specific research questions: 

1. What is the impact of taking online Algebra I for credit recovery on at-risk 9th graders ’ 
short-term academic outcomes (score on end-of- Algebra I course assessment, credit 
attainment, mathematics test scores) and long-term outcomes (geometry and chemistry 
grades; ACT mathematics score; total mathematics and science credits; likelihood of 
dropout in the 2nd, 3rd, or 4th year of high school; 4-year graduation), compared with 
retaking Algebra I as a standard f2f summer course? 

2. How does the instructional experience (engagement, classroom personalism, academic 
demand, self-efficacy in mathematics) compare for students taking online and f2f summer 
credit recovery courses? 
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3. Is online credit recovery differentially beneficial for students with different levels of risk for 
failure (e.g., different failure rates across all courses, different 9th grade GPAs, different 
incoming skills as measured on prior tests)? 

2) To determine the supporting conditions in the classroom under which online Algebra I 
for credit recovery yields higher efficacy. This study will identify specific contributing factors 
to program effectiveness (e.g., mentor practices, online teacher interactions): 

4. Under what supporting conditions in the classroom does the implementation of online 
Algebra I for summer credit recovery produce the strongest treatment effects, relative to f2f 
Algebra I for summer credit recovery? 

3) To gauge the extent to which credit recovery can help at-risk students get back on track, 
relative to students who passed Algebra I in 9th grade. This question will provide critical 
information about whether summer credit recovery options can help close the gap between 
students who failed Algebra I in 9th grade and students who passed: 

5. To what extent do students who take online Algebra I and f2f Algebra I for credit recovery 
“close the gap ” with students who passed Algebra I in 9th grade, in terms of short-term 
outcomes (number of credits, algebra skills) and long-term outcomes (dropout rates over the 
next 3 years of high school, grades in geometry and chemistry)? 

4) To gauge the effects of expanding summer credit recovery through online courses. This 
aim is to assess the effects of expanding credit recovery through expanded online opportunities, 
relative to typical summer credit recovery opportunities: 

6. To what extent does expanded summer credit recovery through online classes lead to 
improved academic outcomes (progress in the mathematics sequence, staying in school, 
improvements in mathematics scores), compared with the outcomes of students in schools 
with limited access to summer credit recovery? 

7. Are the benefits of expanded summer credit recovery options differen t for studen ts with 
different levels of risk for failure (e.g., different failure rates across all courses, different 9th 
grade GPAs, different incoming skills as measured on prior tests)? 

Setting: 

The setting will be CPS high schools with freshman Algebra I failure rates of 20% or higher. 

CPS is the third-largest U.S. district, serving more than 435,000 students in 666 schools, of 
which 1 16 are public high schools and 27 are public charter high schools. School refonn and 
improvement have been high priorities in Chicago for a number of years, as high schools in CPS 
continue to struggle with low student performance and low graduation rates (Kahne, Sporte, de la 
Torre, & Easton, 2006). The overall graduation rate in the district is just 54%. The average 
composite ACT score for CPS juniors is 17, lower than the 20.5 for juniors in the state of Illinois 
and well below the score required by most colleges (Allensworth, Correa, & Ponisciak, 2008). 

Population / Participants / Subjects: 

The target students for this study are first-time freshmen who failed Algebra IB but passed the 
first semester. In CPS schools with freshman failure rates above 20%, the average number of 9th 
graders is 275, and 35% of them fail Algebra IB. We estimate that approximately 25-30% of 
first-time freshmen fail Algebra IB after passing IA (70-83 students per school). We will target 
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these students for recruitment. Our goal is to enroll, on average, 40 students per school in 
summer credit recovery for Algebra IB. 

Intervention / Program / Practice: 

A number of providers offer online courses for credit recovery, including commercial providers 
and courses offered as part of district or state virtual schools programs (Archambault et ah, 

2010). The Department of Graduation Pathways at CPS began to pilot online credit recovery 
options in summer 2008. In the pilot, the district compared two vendors that offered online 
courses for the express purpose of credit recovery. Twenty-one schools participated in the pilot. 
More than 700 students initially registered for an online course in a variety of subjects, and 365 
students attended the full session. Based on review of the rates of successful course completion 
and surveys of mentors and students, CPS discerned that one of the two course providers 
appeared to better serve at-risk students. Specifically, CPS observed that the success (i.e., 
completion) rate for Aventa online summer courses was 72% (vs. 43% for the other provider, 
Apex Learning) and that the Aventa teachers appeared to have been better trained to work with 
at-risk students. The district, mentors, and students noted that the Aventa teachers were more 
communicative, responded more quickly, and gave more support than the teachers in the other 
online course. Since the 2008 pilot, Aventa has been providing credit recovery courses to an 
increasing number of schools and students in CPS. 

Research Design: 

The proposed Goal 3 trial of online Algebra I for summer credit recovery will rely on with in - 
school randomization of struggling 9th graders to condition: online Algebra I (treatment) and 
standard f2f Algebra I (control). We have conducted detailed power calculations to justify the 
size and scope of this student-level random assignment design. 

The within-school basic design will involve 20 public high schools in CPS. We will select the 
schools from a sampling pool of high schools that have freshman Algebra I failure rates above 
20%, based on the previous year’s district records, and that do not have existing expanded 
summer credit recovery programs in place (i.e., schools that do not have external funding to 
increase the number of Algebra I course offerings to increase enrollments of students in need of 
credits). We will focus on these schools because they are in the greatest need of improved 
options for credit recovery and because the high numbers of freshmen who fail the spring 
semester of Algebra I will ensure a sufficient number of students in the two conditions to test our 
central hypotheses. 

Within each school, we will randomly assign students who (a) failed second semester Algebra I, 
(b) are willing to enroll in summer credit recovery, and (c) are willing to take an online course if 
available. Our focus is on students who failed the second semester only and are positioned to get 
back on track in mathematics if they recover the Vi credit in the summer after 9th grade. Our 
estimates of the number of students per class are based on typical enrollment patterns in CPS 
during the 2008 and 2009 summer sessions, with an assumed additional 20% enrollment from 
usual due to a concerted push to motivate students to attempt summer credit recovery in the 
schools participating in the study. For each of the two summers of implementation (2011 and 
2012), we anticipate that we will randomly assign an average of 40 students per school into 
online and f2f classes of 20 students each, for a total of 800 students per cohort. 

Minimum detectable effect sizes (MDESs) for impacts on student achievement and other 
outcomes range from 0.14 to 0.19, for analyses conducted separately by cohort. We believe these 
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MDESs are reasonable given our theory of action that proposes that the online course is more 
engaging for at-risk students, who are thus more likely to persist and complete than students who 
retake the traditional course in which they were previously unsuccessful. These MDESs are also 
reasonable based on previous research that finds the average effect of online learning relative to 
Of is about 0.24 standard deviations (U.S. Department of Education, 2009). This within-school, 
student-level randomized design establishes the core comparison that allows us to answer 
Research Questions 1-4, delineated under Aim 1 and Aim 2. 

Data Collection and Analysis: 

To fully capture the extent to which participation in credit recovery courses affects the future 
perfonnance of students in mathematics, we will collect both achievement and course-taking 
data for both cohorts of high school students. We will also collect annual data on students’ 
dropout/persistence status and, for Cohort 1, graduation status. Of great benefit to this study, 
nearly all of the student outcomes are available for all CPS students; therefore, we will have 
fully intact ITT samples with little missing data. We will also be able to readily obtain, without 
going directly to non-participating schools, all the data required for the secondary comparisons 
(i.e., to on-track and other off-track students with limited access to summer credit recovery). 

Given the nested data structure (i.e., students nested within schools), the primary analytic method 
for this study will be hierarchical linear modeling (HLM; Raudenbush & Bryk, 2002). By 
explicitly taking into account the dependence or clustering among individuals nested within the 
same higher-level units, the HLM method generates more-accurate standard errors for parameter 
estimates and enables researchers to address questions that cannot be addressed with the 
traditional regression model (e.g., heterogeneity of regression slopes). For all analyses, we will 
include the full ITT sample of online and f2f students. We will also conduct exploratory TOT 
analyses to supplement the ITT findings. Further, we will verify the findings from the HLM 
models through fixed-effects models that remove all school-specific variation. 

Findings / Results: 

The study is first being implemented in summer 2011, and we will describe the study design and 
report on the implementation of the first summer cohort, including challenges and lesson learned 
from expanding access to credit recovery courses for at-risk students and conducting random 
assignment -on the spot” as students show up to take summer classes. The paper will also 
describe methods for measuring student participation and engagement in online courses, 
including interactions with online teachers, online students and in-class mentors. 

Conclusions: 

Most of the conclusions will be based on the implementation of the first cohort of students in 
summer 2011. However, the study team has already identified and addressed a number of design 
challenges associated with student-level random assignment that are likely to be highly relevant 
to researchers attending the SREE conference. For example, if a study design calls for on-site 
random assignment that involves stratification and occurs in a large number of schools, what are 
the best processes to ensure that the random assignment will be done correctly and efficiently? 
This issue specifically and other issues surrounding student-level random assignment will be 
presented. 
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